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Timeliness  Matters 
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A  field  experiment  was  conducted  In  which  the  time  period  was 
varied  between  completion  of  the  Skill  Qualification  Test  and  feed¬ 
back  of  results.  Data  are  presented  which  show  the  marked  decline 
In  soldiers'  perceived  utility  of  the  feedback  over  time.  In  addition 
soldiers  Indicate  a  greater  tendency  to  put  off  utilization  of  the 
feedback  the  longer  the  feedback  Is  delayed. 
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Background  and  Purpose. 


Between  15  July  and  30  November  1980,  the  Army  Training  Board 
(ATB)  locally  scored  the  Skill  Qualification  Tests  (SQT)  taken  by 
soldiers  at  Ft  Bragg,  NC.  Normally,  these  tests ^would  have  been  shipped 
to  the  central  scoring  facility  at  Ft  Eustls,  VA,  and  the  results  shipped 
back  to  Ft  Bragg,  The  Local  Scoring  Evaluation  (LSE)  project  was  carried 
out  to  determine  whether  the  advantages  of  local  scoring  are  worth  the 
additional  costs  and  efforts  such  a  system  requires.  This  paper  des¬ 
cribes  a  portion  of  the  research  conducted  by  the  Army  Research  Institute 
(ARI)  in  support  of  the  LSE  project. 

A  major  objective  of  ARI's  technical  advisory  service  (TAS)  to  the 
Army  Training  Board  during  the  Local  Scoring  Evaluation  was  to  determine 
whether  the  speed  of  feedback  of  SQT  results  had  any  Immediate  effects 
on  individual  soldiers.  The  existing  centralized  scoring  system  had  a 
goal  of  thirty  days*  turnaround  time  for  SQT  results.  Although  this 
seemed  Inordinately  long,  no  evidence  existed  to  indicate  whether  any 
particular  number  of  days  should  be  preferred.  Given  that  immediate 
feedback  was  not  possible,  the  question  became  whether  there  was  any 
practical  turnaround  time  that  might  make  a  difference  to  soldiers. 

To  answer  this  question,  a  field  experiment  was  set  up  In  conjunc¬ 
tion  with  the  LSE  project.  The  basic  idea  was  to  vary  the  length  of 
time  between  completion  of  the  SQT  and  feedback  of  results,  and  to  ob¬ 
tain  the  soldiers'  reactions  to  the  variation. 


Design  Plan 

The  design  plan  divided  the  82d  Airborne  Division  Into  three  approxi¬ 
mately  similar  groups  and  assigned  each  to  one  of  three  treatment  conditions 
The  treatment  variable  was  speed  of  SQT  results  feedback.  Its  three  levels 
were  Fast,  Medium,  and  Slow.  "Fast"  meant  that  a  soldier  in  the  group  re¬ 
ceived  the  results  of  his  SQT  within  1  to  15  days;  "Medium"  16  to  30  days; 
"Slow"  31  to  45  days.  The  Slow  condition  was  believed  to  simulate  the 
current  central  system's  normal  operation. 

Composition  of  the  groups  by  unit  was  the  following: 


FAST 

2d  Bn  (ABN) ,  508th  IN 
3d  Bn  (ABN),  325th  IN 
2d  Bn  (ABN) ,  505th  IN 
7 St  Bn  (ABN),  319th  FA 
782  Maint  Bn 
82d  Finance  Co 
1st  Squadron,  17th  Cav 
82d  Signal  Bn 
82d  MP  Co 


MEDIUM 

2d  Bn  (ABN) ,  504th  IN 
2d  Bn  (ABN),  325th  IN 
1st  Bn  (ABN),  505th  IN 
1st  Bn  (ABN),  320th  FA 
407th  S&S  Bn 
82d  AG  Co 
82d  Aviation  Bn 
4th  Bn,  68th  Armor 
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SLOW 


1st  Bn  (ABN),  504th  IN 
1st  Bn  (ABN),  325th  IN 
1st  Bn  (ABN),  508th  IN 
2d  Bn  (ABN),  321st  FA 
307th  Med  Bn 
2l8t  Chem 

3d  Bn  (ABN),  4th  AD  Arty 
313th  MI  Bn  (CEWI) 

307th  Eng  Bn 

The  formation  of  the  groups  was  made  with  the  following  rules  In 
mind:  (a)  Each  group  should  be  about  the  same  size,  (b)  The  MOS  In¬ 
volved  should  be  locally  scored,  (c)  The  MOS  distribution  across  groups 
should  be  balanced,  (d)  For  a  brigade  composed  of  three  battalions,  one 
battalion  should  be  assigned  to  each  group.  The  goal  of  the  design  plan 
was  to  define  three  comparable  groups  from  which  samples  could  be  obtained 
and  analyzed.  In  a  later  section  the  samples  will  be  described. 

Instruments 

The  form  of  the  SQT  results  feedback  was  a  computer  printout  called 
the  Individual  Soldier  Report  (ISR)  which  gave  the  soldier's  SQT  score, 
a  detailed  list  of  failed  tasks,  failed  Items  within  tasks,  and  references 
to  paragraphs  In  Soldier's  Manual.  The  ISRs  were  generated  by  ATB'a 
local  scoring  facility  at  Ft  Bragg. 

The  Instrument  used  to  obtain  soldiers'  reactions  to  the  feedback  of 
their  SQT  results  was  the  ISR/BQr  Feedback  Evaluation  Record.  Questions 
on  the  Evaluation  Record  were  aimed  at  determining  the  utility  of  the 
ISR,  SQT  score,  certain  SQT  preparation  variables,  and  some  Indications 
of  what  these  SQT  results  meant  to  the  soldier.  Try-out  of  this  Instrument 
during  Its  developmental  phase  showed  that  nearly  all  soldiers  could  com¬ 
plete  It  In  less  than  twenty  minutes. 

Procedures 


The  procedures  followed  In  this  research  were  the  following: 

1.  An  ARI  researcher  personally  delivered  or  observed  the 
delivery  of  an  ISR  to  a  soldier. 

2.  The  soldier  was  given  sufficient  time  to  examine  his  ISR 
and  digest  Its  contents. 

3.  An  ARI  researcher  administered  the  ISR/SQT  Feedback 
Evaluation  Record  to  the  soldier. 

This  data  collection  was  accomplished  under  real-time  operational  condition 
and  was  dependent  on  each  unit's  SQT  scheduling  choices.  Thus  It  was 
sometimes  necessary  because  of  manpower  and  time  constraints  to  decide 
which  units  to  sample.  High  density  MOS  were  preferred  over  low  density 


MOS,  large  units  over  small  ones,  and  combat  MOS  over  non-combat  MOS. 
Adherence  to  these  priorities  resulted  in  a  fairly  large  total  sample 
with  sufficient  numbers  of  soldiers  at  the  treatment  levels  to  permitf 
statistical  analysis. 

Sample  Description 

During  the  duration  of  the  LSE  at  Ft  Bragg  3166  SQTs  were  scored 
and  their  corresponding  ISRs  produced.  A  total  of  576  soldiers  completed 
the  ISR/SQT  Feedback  Evaluation  Record  and  558  of  these  were  usable  in 
this  analysis.  As  might  be  expected  from  the  procedural  priorities,  the 
combat  MOS*  provided  more  than  their  share  of  the  sample.  In  the  analyzed 
sample  513  of  558  or  91.9Z  were  combat  MOS,  whereas  2232  of  the  3166 
SQTs  scored  or  70. 5Z  were  in  combat  MOS. 

At  this  point  a  variety  of  reasons  compelled  the  reduction  of  the 
sample  to  be  analyzed  to  Just  the  combat  MOS.  Table  1  shows  one  of  the 
realities  of  coordinating  a  real-time  experiment  under  operational 
conditions:  No  non-combat  MOS  soldiers  were  obtained  under  the  "Slow" 
feedback  condition. 

Table  1.  Number  of  soldiers  at  each  treatment  level 


Fast 

(1-15  days) 

Medium 
(16-30  days) 

Slow 

(31-45  days) 

N 

Coiibat  MOS 

255 

111 

147 

513 

Non-combat  MOS 

28 

17 

0 

45 

Total 

283 

128 

147 

558 

Tables  2  and  3  show  the  percentage  distributions  of  the  combat  and  non¬ 
combat  MOS  on  two  background  variables  which  might  Interact  with  reactions 
to  SQT  results  feedback,  time  in  service  and  time  left  to  serve.  The 
dissimilarities  between  conbat  and  non-combat  MOS  soldiers  appear  to 
further  Justify  removing  the  few  non-combat  MOS  soldiers  from  the  analysis. 


Table  2.  Time  in  service  (months)  percentage  distribution 


12  or  less 

13-24 

25-36 

37-48 

More  than  48 

Combat  MOS 

(N-513) 

4.1 

50.5 

19.1 

13.3 

13.1 

Non-combat 

MOS  (N-45) 

0.0 

46.7 

26.7 

0.0 

26.7 

*Combat  MOS  were  CMF  11,  12,  13,  17  or  19;  non-combat  MOS  were  any  others. 
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Table  3.  Time  left  to  serve  (months)  percentage  distribution 


12  or  less 

13-24 

25-36 

37-48 

More  than  48 

Coiid}at  MOS 

(N-513) 

25.7 

39.8 

28.5 

3.7 

2.3 

Non-conbat 

MOS  (N-45) 

28.9 

57.8 

8.9 

0.0 

4.4 

Thus  a  fairly  large  (N*513)  and  fairly  homogeneous  (all  combat  MOS) 
sample  was  constructed  for  determining  whether  Increasing  the  speed 
of  feedback  of  SQT  results  makes  any  difference  to  soldiers. 

Results 


Of  primary  concern  to  ATB  was  whether  Increasing  the  speed  of  SQT 
results  feedback  mattered  or  had  any  effect  on  soldiers. 

A  question  on  the  ISR/SQT  Feedback  Evaluation  Record  addressed  this 
directly,  l.e.,  "How  useful  to  you  Is  this  ISR  to  the  Improvement  of 
your  military  skills?"  Responses  were  obtained  on  a  five  point  scale: 
5«very  useful,  4*useful,  3*sll^tly  useful,  2«not  very  useful,  and  l"ln 
no  way  useful.  The  means  of  the  conbat  MOS  soldiers  for  the  three 
treatment  levels  are  shown  In  Table  4. 

Table  4.  Ratings  of  ISR  usefulness  by  soldiers  with  coDd>at  MOS 


Speed  of  SQT  Feedback 

Mean 

S.D. 

N 

Fast  (1  to  15  days) 

4.27 

.89 

253 

Medium  (16  to  30  days) 

3.25 

1.49 

111 

F^60.24 

Slow  (31  to  45  days) 

2.62 

1.34 

146 

p  <  .001 

3.57 

1.38 

510 

Table  4  Indicates  quite  clearly  that  soldiers  had  a  more  favorable  view 
of  the  feedback  of  SQT  results  the  earlier  It  came.  Another  variable 
which  might  have  affected  a  soldier's  Inmedlate  reactions  to  feedback 
was  whether  It  was  positive  or  negative,  did  he  pass  or  fall.  Tables  5 
and  6  tdilch  separate  passers  and  fallers  show  the  same  pattern  as  before 
with  a  slight  decrease  In  overall  mean  usefulness  rating  by  the  fallers. 


Table  5.  Mean  ratings  of  ISR  usefulness  by  soldiers  with 

combat  MOS  who  passed  SQT  (SQT  score  of  60  or  more) 


Speed  of  SQT  Feedback 

Mean 

S.D. 

N 

Fast  (0-15  days) 

4.28 

0.87 

240 

Medium  (16-30  days) 

3.22 

1.50 

92 

FW91.87 

Slow  (31  to  45  days) 

2.62 

3.59 

1.36 

1.38 

134 

466 

p  <  .001 
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Table  6.  Mean  ratings  of  ISR  usefulness  by  soldiers  with  combat 


MOS  who  failed  SQT 

(SQT  score  of  less 

than  60) 

Speed  of  SQT  Feedback 

Mean 

S.D. 

N 

Fast  (1-15  days) 

4.15 

1.17 

13 

Medium  (16-30  days) 

3.42 

1.39 

19 

F-4.22 

Slow  (31-45  days) 

2.67 

1.03 

12 

p  <  .05 

3.43 

1.36 

44 

The  gradient  of  feedback  utility  shown  In  these  results  Is  useful 
to  people  who  must  design  the  data  processing  and  distribution  system. 

For  people  concerned  with  training,  however,  there  Is  the  question  of 
what  action  will  take  place  as  a  consequence  of  providing  timely  and 
useful  feedback.  Some  questions  were  asked  on  the  Evaluation  Record 
In  order  to  obtain  a  notion  of  what  the  soldiers  would  do.  Results  of 
two  questions  are  shown  on  Table  7.  One  question  asks  the  soldier  when 
he  expects  to  study  on  his  own  the  Items  he  missed;  the  other  asks  when 
he  believes  his  unit  should  provide  training  on  what  was  missed.  The 
a  priori  best  answer,  l.e.,  a  five  on  a  five  point  scale.  Is  "In  the 
next  few  days."  The  expectation  might  be  that  fallers  would  have  lower 
means  than  passers  for  both  questions,  and  they  did.  What  Is  Interesting, 
however.  Is  that  both  groups  look  to  their  unit  to  move  more  quickly  on 
taking  remedial  action  than  they  Intend  themselves. 


Table  7.  Mean  response  of  combat  MOS  soldiers. 

(5* In  the  next  few  days,  4«next  month,  3* In  the  next  six  months 
2*just  before  the  next  SQT,  and  l^don't  know) 


Question 

Passers 
Mean  N 

Mean 

Fallers 

N 

When  do  you  expect  to  do  some 
training/studying,  on  your  own 
on  the  task  Items  you  missed? 

2.8 

466 

2.5 

45 

When  do  you  believe  your  unit 
should  provide  some  training 
on  the  tasks  most  soldiers 
failed? 

3.1 

466 

2.8 

45 

These  questions  were  also  analyzed  In  terms  of  the  speed  of  feedback 
variable.  Shown  In  Table  8  are  the  percentages  of  soldiers  responding 
"Don't  Know"  to  the  two  questions.  There  are  two  important  points  to  be 
made  In  regard  to  these  results.  One  Is  that  these  are  rather  substantial 
percentages*  Perhaps  this  Is  a  true  picture  as  far  as  the  soldier's  own 
need  or  opportunity  to  self-train.  The  other  point  Is  that  the 
percentage  of  Don't  Knows  Increases  as  the  time  since  SQT  Increases.  It 
could  be  suggested  here  that  the  longer  the  delay  between  testing  and 
feedback  of  results,  the  more  depressed  Is  the  motivation  to  review  or 
take  any  remedial  actions. 
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Table  8.  Percentages  of  soldiers  with  combat  MOS  responding  "Don't  Know" 


When  do  you  expect  to  do  25.9%  27.0%  32.0% 

some  training/studying, 
on  your  own,  on  the  task 
Items  you  missed? 


When  do  you  believe  your  14.1%  18.0%  18.4% 

unit  should  provide  some 
training  on  the  tasks  most 

soldiers  failed?  _  _  _ 

N-255  N-111  N-147 


Discussion 

In  learning  psychology,  feedback  Is  viewed  as  a  secondary  relnforcer 
and  as  such  has  Its  greatest  effect  when  It  Is  Immediate.  With  soldiers 
and  their  SQT  performance  It  is  currently  not  possible  to  provide  Immediate 
feedback.  It  Is,  however,  possible  to  significantly  Increase  the  speed 
of  feedback  from  a  matter  of  several  weeks  to  a  few  days.  In  this  research 
It  was  found  that  even  though  the  speed  of  feedback  falls  far  short  of 
being  Immediate,  speed  still  matters.  The  perceived  utility  of  the  feed¬ 
back  (the  ISR)  Is  a  function  of  time,  remaining  hl^  during  the  1-15  day 
period  and  then  dropping  steadily  to  a  point  where  its  mean  rating  Is 
less  than  "slightly  useful"  somewhere  between  31  and  45  days  after  SQT 
completion.  This  pattern  was  the  same  for  both  fallers,  where  the  feed¬ 
back  had  a  negative  message,  and  passers  where  the  feedback  was  relatively 
positive. 

A  high  value  placed  on  the  utility  of  feedback  does  not,  of  course, 
mean  that  the  Information  supplied  Is  taken  to  heart  and  remedial  studying 
or  training  Is  pursued  forthwith.  Results  of  this  research  Indicate, 
however,  that  soldiers  who  received  their  ISRs  more  quickly  Intend  to 
do  some  studying  or  training  sooner  than  those  whose  ISRs  were  less 
timely.  Spee&  of  feedback  was  also  associated  with  the  desire  to  have 
sooner  unit  training  on  items  missed  on  the  SQT.  It  ml^t  be  hypothesized 
that  the  speed  with  which  SQT  results  were  fed  back  to  soldiers  had  a 
pacing  effect,  l.e.,  rapid  feedback  suggested  that  rapid  corrective 
action  was  expected  whereas  slow  feedback  suggested  there  was  no  urgency 
about  correcting  mistakes. 

The  Increase  over  time  of  "Don't  Know"  responses  to  the  questions 
of  when  studying  or  training  will  or  should  be  done  could  have  several 
causes.  What  seems  most  likely  Is  that  Interest  or  motivation  had 
waned  with  the  passage  of  time.  SQT  was  a  hot  topic  for  a  week  or  two 
and  then  became  another  event  to  be  forgotten.  It  could  also  be  that 
for  many  soldiers  there  became  objectively  less  reason  to  be  concerned 
about  their  ability  to  perform  the  SQT  tasks.  Whatever  the  reason.  It 
appeared  that  the  longer  the  feedback  of  results  was  delayed,  the  less 
likely  soldiers  would  be  to  constructively  use  Information  contained 
In  the  feedback. 
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Conclusion 


SQT  feedback  In  the  form  of  the  ISR  has  a  rather  short  shelf  life, 
perhaps  two  weeks.  The  data  presented  show  a  noticeable  decline  of 
the  perceived  utility  of  the  ISR  over  time.  There  are  also  indications 
that  the  longer  the  delay  In  getting  the  feedback  to  the  soldier,  the 
weaker  Is  his  Intention  to  act  on  It.  The  results  further  show  that 
the  soldier  expects  more  Initiative  from  his  unit  than  from  himself 
when  It  comes  to  training  on  SQT  tasks  which  were  failed. 


